(1) Field of the Invention
The present invention relates to an image processing method and an image processing device, and more particularly to a technology of reconstructing a high-resolution picture from captured low-resolution pictures.
(2) Description of the Related Art
In recent years, digital video cameras have been widely used to capture images in daily life. However, in the capturing, hands cannot hold the cameras still so that the captured images often have degraded quality. There are two types of methods for modifying (hereinafter, referred to also as “stabilization” or “modifying instability”) the instability of the camera capturing: an optical method using known mechanical processing; and an electronic method using electrical processing. Since the optical method has disadvantages of difficulty in miniaturization and cost reduction, the electronic method is often utilized. One conventional electronic stabilization method is disclosed in Japanese Patent Application Publication No. 2005-130159, for example. In this conventional method, deviation (hereinafter, referred to also as “motion” or a “motion vector”) between temporally neighbor pictures is calculated to detect an in-stabilized component, and each pixel in a target picture is modified by eliminating the in-stabilized component. In short, by shifting the entire target picture with the motion vector, the instability is modified.
However, in general cameras, resolution of images outputted from imaging sensors is low, and a target picture for which instability is to be modified is often out of focus. Therefore, a problem is that, even if instability of the target picture is modified, the resulting picture turns out to still have low resolution failing to enhance the resolution. Furthermore, if image modification, such as affine transformation, is performed for the stabilization of image signals, it is necessary to obtain pixel values using linear interpolation. Thereby, another problem is that the images stabilized in the above manner turn out to be blurred. Still further, when pixels whose values are obtained by the linear interpolation and pixel whose values are not obtained by the linear interpolation are mixed spatially and temporally, the resulting pixels are a mix of blurred pixels and not-blurred pixels. This causes still another problem of subjective degradation of the image quality. Here, the linear interpolation processing is not performed when the stabilization is processed in units of integer-numbered pixels. On the other hand, the linear interpolation processing is performed when the stabilization is processed not in units of integer-numbered pixels, in other words, processed in units each of which is smaller than one pixel (with sub-pixel precision).
In the meantime, another image processing method, a so-called super-resolution image processing method, has recently been proposed in, for example, “Super-Resolution Image Reconstruction: A Technical Overview”, S. C. Park, M. K. Park, and M. G. Kang, “IEEE Signal Processing Magazine”, May 2003. In this super-resolution image processing method, a sequence of pictures included in low-resolution video is integrated to reconstruct a high-resolution picture.
In more detail, each deviation amount (motion amount) between low-resolution pictures is detected with sub-pixel precision. Then, according to the deviation amount, the integration of the low-resolution pictures are executed.
Here, it is conceivable that such super-resolution image processing is performed after stabilization. That is, there is a case where pixels in a picture is applied with both of the above-explained stabilization processing and this super-resolution image processing, in other words, where a high-resolution picture is generated after stabilizing image signals in the low-resolution pictures.
In such a case, however, it is necessary to calculate respective overlapping deviation amounts between pictures, for both of the stabilization processing and the super-resolution image processing, which results in still another problem of increase of a processing amount and a circuit size.